Introduction to China’s CWMT2008 Machine Translation Evaluation

This paper presents an overall introduction to the CWMT2008 evaluation and focuses on its two new metrics: BLEU-SBP (Chiang et al., 2008) and linguistic check-point method (Zhou et al., 2008). BLEU-SBP is a revised BLEU with strict brevity penalty. Our experiments validated BLEU-SBP’s effectivity in resolving the nondecomposability problem of both NIST-BLEU and IBMBLEU at sentence level. Linguistic checkpoint method (LCM) is a linguistic diagnostic evaluation method based on automatically constructed linguistic check-points, and our evaluation indicates that this method can be used to evaluate the capability of an MT system in translating various linguistic phenomena, and revealed good correlations between BLEU score and LCM scores in most tasks. With the aid of these metrics, we disclosed some detailed performance differences between statistical MT systems and rulebased MT systems. In addition, through the study on some practical cases we suggest that the high BLEU score doesn’t necessarily mean high translation adequacy.